AMENDMENTS TO THE SPECIFICATION 
Please amend the paragraph beginning on page 8, line 24, as follows: 
In one embodiment, the implicit relevance data 204 may be collected for a single 
interaction, aggregated across all interactions in a given user session of interacting with search 
results, or further aggregated across all users' interactions with the same search results during 
their own similar searches. For example, a basic unit of implicit relevance data 204 is an atomic 
measurement of one user, one query, and one result interaction. There may be several atomic 
measurements for a given session, such as printing a result, bookmarking the result in a favorites 
folder, sending the result in an e-mail to some friends, etc. By aggregating the atomic 
measurements for all users and all queries, the implicit relevance data 204 is a sufficiently large 
and detailed corpus of data that is an excellent predictor of relevance. In a preferred 
embodiment, the automated relevance optimization system 200 collects implicit relevance 
data 204, as described in detail in commonly assigned copending United States Patent 

Application No. [[ H 10/805,873 , which is herein incorporated by reference. Other 

methods of collecting implicit relevance data 204 may be implemented without departing from 
the scope of the claims that follow, as long as the data is sufficiently large and detailed to be 
highly predictive of search result relevance. 

Please amend the paragraph beginning on page 16, line 3, as follows: 
FIGURES 5A-5B are flow diagrams illustrating the logic performed in conjunction wi t h 
the search engine -s erver 1 12 and automated rel evance optimizer 202 of FIGURES 2 and 3 for by 
a relevance optimizer method 500 for automating the optimization of search results in a search 
engine in accordance with an embodiment of the present invention. The search engine 
server 112 begins at the start block 502 and continues at processing block 504 to generate the 
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search Web page 106 with search results 110 (FIGURE 1) in accordance with the existing 
relevance schema database 114. At processing block 506, the automated relevance optimizer 
process 202 collects the relevance performance data for the search results 110 from various 
sources, including the implicit relevance data 204, the explicit relevance data 206, the human- 
judged test data 208, the relevance verification test data 210, and the sample A/B test data 21 2. 
In one embodiment, collecting the implicit relevance data 204 may include aggregating the data 
across a local user's session or across multiple sessions, or even across multiple users where 
cross-user data is available from the search engine server 112. Alternatively, the implicit 
relevance data 204 may be already aggregated by the search engine server 112 for use by the 
automated relevance optimizer process 202. In any event, processing continues at process 
block 508, where the automated relevance optimizer process 202 compiles the various sources of 
collected relevance performance data into a measurement of the actual performance of a result or 
results under consideration, including normalizing the various sources of performance data in 
accordance with their relative importance. In one embodiment, the relative importance of the 
various sources is a measure of their value in predicting the relevance of search results, and that 
value may be predefined by the search engine operator, and further changed from time to time to 
aid the operator in fine-tuning the automated relevance optimization process. In a preferred 
embodiment, the value of the implicit relevance data 204 is likely to be higher than other sources 
of relevance performance data because it may be highly predictive due to the potentially large 
scale of data collection and rapidity with which the data may be collected . 

Please amend the paragraph beginning on page 16, line 29, as follows: 
In one embodiment, the automated relevance optimizer process 202 method 500 
continues at process block 510 to obtain the expected relevance data 214 for the search result or 
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results under consideration. The expected relevance data 214 may be predefined by the search 
engine operator and changed from time to time to reflect changes in the expected performance of 
search results over time. In one embodiment, the expected relevance data 214 may have even 
been changed or otherwise updated by an action 216 generated by a previous iteration of the 
automated relevance optimizer 202 to reflect any automated changes in expectations. 

Please amend the paragraph beginning on page 17, line 7, as follows: 
In one embodiment, the automated relevance optimizer p rocess 202 method 500 
continues at process block 5 12 to compare the measurement of the actual performance of a result 
or results under consideration to the expected performance, where the actual perfonnance was 
determined at process block 508, and the expected perfonnance was determined at process 
block 510. When the comparison is unfavorable, e.g., the actual performance fall short or 
substantially short of the expected performance, the search results are underperforming, which 
may indicate a problem with the relevance of the search results. The automated relevance 
optimizer process attempts to diagnose the possible cause or causes of the problem. For 
example, in some cases the result is obsolete and other newer results are now more relevant, as 
reflected, for example, in the implicit relevance data 204 collected for the results. In other cases, 
the search term for which the search results were generated is too broad or easily misspelled, and 
requires the search engine to prompt the user to clarify the terms. Numerous other diagnoses of 
possible causes of the problem may be made without departing from the scope of the claims that 
follow. 
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Please amend the paragraph beginning on page 17, line 21, as follows: 
In FIGURE 5B, the automated relevance optimizer 302 method 500 continues at decision 
block 514 to determine whether a problem with the relevance of the search results has been 
diagnosed. If not, the automated relevance optimizer 202 ends at termination oval 520. 
Otherwise, processing continues at process block 516, where the automated relevance 
optimizer 202 determines what adjustment or corrective action 216 (FIGURE 2) to generate in an 
effort to address the problem. As further illustrated in FIGURE 5, the adjustment or corrective 
action 216 is applied at block 518. As described earlier, the action 216 may include adjustments 
to the search engine operation that cause the user to be prompted to clarify or narrow the search 
term in cases where the problem is related to the search term being too general or easily 
misspelled. In one embodiment, the action 216 may include a modification, either temporary or 
permanent, to the relevance schema database 114, or even modifications to the expected 
relevance data 214. In still another embodiment, the action 216 may comprise one or more 
modifications to the search results 108 on the search Web page 106 in real time, where the search 
results 1 10 that were generated by the search engine 1 12 are intercepted and reranked, reordered, 
reformatted, removed, replaced with other results, or otherwise modified in an effort to optimize 
the search result relevance for the user. Combinations of above-described actions 216 may be 
employed as well without departing from the scope of the claims that follow. 

Please amend the paragraph beginning on page 1 8, line 8, as follows: 
While the presently preferred embodiments of the invention have been illustrated and 
described, it will be appreciated that various changes may be made therein without departing 
from the spirit and scope of the invention. For example, as already described, in one 
embodiment of the present invention, the automated relevance o ptimization system pr oc ess 202 
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optimizer method 500 and associated subprocesses for data collection 310, diagnosis 3 1 2, and 
adjustment 314, may be implemented in real time to allow for up-to-the-minute optimizations 
based on the latest performance data captured by the search engine server 1 12 and collected by 
the automated relevance optimizer 202. In another embodiment, the automated relevance 
optimization system 200 optimizer method 500 processes may be implemented in batch mode to 
allow for data collection of performance data from a variety of sources, including implicit 
relevance data 204, explicit relevance data 206, human-judged test data, relevance verification 
test data 2 10, and sample A/B test data 212, and a combination of automated and manual 
optimizations of search results. In yet other embodiments, the automated optimization search 
result relevance system 200 optimizer method 500 may be limited in application to consideration 
of less than all sources of performance data, e.g., limited to the implicit relevance data 204, as 
well as limited in application to only certain types of actions 216, such as permanent 
modifications to the relevance schema database 1 14, real time updates to the relevance schema 
database 1 14 or to the search results 1 10, or any combination thereof. 
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